Acoustic transfer function personalization using sound scene analysis and beamforming

ABSTRACT

An audio system for a wearable device dynamically updates acoustic transfer functions. The audio system is configured to estimate a direction of arrival (DoA) of each sound source detected by a microphone array relative to a position of the wearable device within a local area. The audio system may track the movement of each sound source. The audio system may form a beam in the direction of each sound source. The audio system may identify and classify each sound source based on the sound source properties. Based on the DoA estimates, the movement tracking, and the beamforming, the audio system generates or updates the acoustic transfer functions for the sound sources.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. application Ser.No. 16/379,450, filed Apr. 9, 2019, which is incorporated by referencein its entirety.

BACKGROUND

The present disclosure generally relates to sound scene analysis, andspecifically relates to using system feedback to improve sound sceneanalysis.

A sound perceived at two ears can be different, depending on a directionand a location of a sound source with respect to each ear as well as onthe surroundings of a room in which the sound is perceived. Humans candetermine a location of the sound source by comparing the soundperceived at each ear. In a “surround sound” system, a plurality ofspeakers reproduce the directional aspects of sound using acoustictransfer functions. An acoustic transfer function represents therelationship between a sound at its source location and how the sound isdetected, for example, by a microphone array or by a person. A singlemicrophone array (or a person wearing a microphone array) may haveseveral associated acoustic transfer functions for several differentsource locations in a local area surrounding the microphone array (orsurrounding the person wearing the microphone array). In addition,acoustic transfer functions for the microphone array may differ based onthe position and/or orientation of the microphone array in the localarea. Furthermore, the acoustic sensors of a microphone array can bearranged in many possible combinations, and, as such, the associatedacoustic transfer functions are unique to the microphone array. As aresult, determining acoustic transfer functions for each microphonearray can require direct evaluation, which can be a lengthy andexpensive process in terms of time and resources needed.

SUMMARY

An audio system for a wearable device dynamically updates acoustictransfer functions. The audio system is configured to estimate adirection of arrival (DoA) of each sound source detected by a microphonearray relative to a position of the wearable device within a local area.The audio system may track the movement of each sound source. The audiosystem may isolate the signal from each sound source. The audio systemmay identify and classify each sound source based on the sound sourceproperties. Based on the DoA estimates, the movement tracking, and thesignal isolation, the audio system generates or updates the acoustictransfer functions for the sound sources.

Systems, methods, and articles of manufacture for dynamically updatingacoustic transfer functions are disclosed. In some embodiments, therecited components may perform actions including: detecting, via amicrophone array of a wearable device, sounds from one or more soundsources in a local area of the wearable device; estimating acoustictransfer functions associated with the sounds; estimating a direction ofarrival (DoA) of a sound source in the one or more sound sources;tracking a movement of the sound source; and updating the acoustictransfer functions based on the movement of the sound source.

In various embodiments, the sound source may be classified based on aclassification library. The signal from the sound source may be isolatedfrom other sound sources in the local area of the wearable device. Afirst confidence level for the tracking, a second confidence level forthe classifying, and a third confidence level for a beamforming processmay be calculated. The acoustic transfer functions may be updated basedon at least one of the first confidence level, the second confidencelevel, or the third confidence level. The tracking may comprise storingvalues for the number and locations of the one or more sound source overtime, and detecting a change in at least one of the number or thelocations. The system may update sound filters based on the updatedacoustic transfer functions. The system may present audio content basedon the updated sound filters.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a wearable device, in accordance with one or moreembodiments.

FIG. 2A illustrates a wearable device analyzing a sound scene within alocal area, in accordance with one or more embodiments.

FIG. 2B illustrates a wearable device analyzing a sound scene within alocal area after movement of a sound source, in accordance with one ormore embodiments.

FIG. 3 is a block diagram of an example audio system, in accordance withone or more embodiments.

FIG. 4 is a process for analyzing a sound scene, in accordance with oneor more embodiments.

FIG. 5 is a system environment of a wearable device including an audiosystem, in accordance with one or more embodiments.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION

A wearable device may determine personalized acoustic transferfunctions. The determined acoustic transfer functions may then be usedfor many purposes, such as to analyze a sound scene or to generate asurround sound experience for the person. To improve accuracy, multipleacoustic transfer functions may be determined for each speaker location(i.e., each speaker is generating a plurality of discrete sounds) in thewearable device.

An audio system in the wearable device detects sound sources to generateone or more acoustic transfer functions for a user. In one embodiment,the audio system includes a microphone array that includes a pluralityof acoustic sensors and a controller. Each acoustic sensor is configuredto detect sounds within a local area surrounding the microphone array.At least some of the plurality of acoustic sensors are coupled to awearable device, such as a near-eye display (NED) configured to be wornby the user.

The controller is configured to estimate a direction of arrival (DoA) ofeach sound source detected by the microphone array relative to aposition of the wearable device within the local area. The controllermay track the movement of each sound source. The controller may form abeam for each sound source. The controller may identify and classifyeach sound source based on the sound source properties. Based on the DoAestimates, the movement tracking, and the beamforming, the controllergenerates or updates acoustic transfer functions for the sound sources.

An acoustic transfer function characterizes how a sound is received froma point in space. Specifically, an acoustic transfer function definesthe relationship between parameters of a sound at its source locationand the parameters at which the sound is detected by, for example, amicrophone array or an ear of a user. The acoustic transfer function maybe, e.g., an array transfer function (ATF) and/or a head-relatedtransfer function (HRTF). Each acoustic transfer function is associatedwith a particular source location and a specific position of thewearable device within the local area, such that the controller mayupdate or generate a new acoustic transfer function as the position ofthe sound source changes within the local area. In some embodiments, theaudio system uses the one or more acoustic transfer functions togenerate audio content (e.g., surround sound) for a user wearing thewearable device.

Embodiments of the invention may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a virtual reality (VR),an augmented reality (AR), a mixed reality (MR), a hybrid reality, orsome combination and/or derivatives thereof. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,and any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto the viewer). Additionally, in some embodiments, artificial realitymay also be associated with applications, products, accessories,services, or some combination thereof, that are used to, e.g., createcontent in an artificial reality and/or are otherwise used in (e.g.,perform activities in) an artificial reality. The artificial realitysystem that provides the artificial reality content may be implementedon various platforms, including a wearable device connected to a hostcomputer system, a standalone wearable device, a mobile device orcomputing system, or any other hardware platform capable of providingartificial reality content to one or more viewers.

FIG. 1 is an example illustrating a wearable device 100 including anaudio system, according to one or more embodiments. As illustrated, thewearable device 100 may be an eyewear device designed to be worn on ahead of a user. In other embodiments, the wearable device 100 may be aheadset, necklace, bracelet, a clip-on device, or any other suitabledevice which may be worn or carried by a user. The wearable device 100presents media to a user. In one embodiment, the wearable device 100 maycomprise a near-eye display (NED). In another embodiment, the wearabledevice 100 may comprise a head-mounted display (HMD). In someembodiments, the wearable device 100 may be worn on the face of a usersuch that content (e.g., media content) is presented using one or bothlenses 110 of the wearable device 100. However, the wearable device 100may also be used such that media content is presented to a user in adifferent manner. Examples of media content presented by the wearabledevice 100 include one or more images, video, audio, or some combinationthereof. The wearable device 100 includes the audio system, and mayinclude, among other components, a frame 105, a lens 110, and a sensordevice 115. While FIG. 1 illustrates the components of the wearabledevice 100 in example locations on the wearable device 100, thecomponents may be located elsewhere on the wearable device 100, on aperipheral device paired with the wearable device 100, or somecombination thereof.

The wearable device 100 may correct or enhance the vision of a user,protect the eye of a user, or provide images to a user. The wearabledevice 100 may be eyeglasses which correct for defects in a user'seyesight. The wearable device 100 may be sunglasses which protect auser's eye from the sun. The wearable device 100 may be safety glasseswhich protect a user's eye from impact. The wearable device 100 may be anight vision device or infrared goggles to enhance a user's vision atnight. The wearable device 100 may be a near-eye display that producesartificial reality content for the user. Alternatively, the wearabledevice 100 may not include a lens 110 and may be a frame 105 with anaudio system that provides audio content (e.g., music, radio, podcasts)to a user.

The lens 110 provides or transmits light to a user wearing the wearabledevice 100. The lens 110 may be prescription lens (e.g., single vision,bifocal and trifocal, or progressive) to help correct for defects in auser's eyesight. The prescription lens transmits ambient light to theuser wearing the wearable device 100. The transmitted ambient light maybe altered by the prescription lens to correct for defects in the user'seyesight. The lens 110 may be a polarized lens or a tinted lens toprotect the user's eyes from the sun. The lens 110 may be one or morewaveguides as part of a waveguide display in which image light iscoupled through an end or edge of the waveguide to the eye of the user.The lens 110 may include an electronic display for providing image lightand may also include an optics block for magnifying image light from theelectronic display. Additional detail regarding the lens 110 isdiscussed with regards to FIG. 5.

In some embodiments, the wearable device 100 may include a depth cameraassembly (DCA) (not shown) that captures data describing depthinformation for a local area surrounding the wearable device 100. Insome embodiments, the DCA may include a light projector (e.g.,structured light and/or flash illumination for time-of-flight), animaging device, and a controller. The captured data may be imagescaptured by the imaging device of light projected onto the local area bythe light projector. In one embodiment, the DCA may include two or morecameras that are oriented to capture portions of the local area instereo and a controller. The captured data may be images captured by thetwo or more cameras of the local area in stereo. The controller computesthe depth information of the local area using the captured data anddepth determination techniques (e.g., structured light, time-of-flight,stereo imaging, etc.). Based on the depth information, the controllerdetermines absolute positional information of the wearable device 100within the local area. The DCA may be integrated with the wearabledevice 100 or may be positioned within the local area external to thewearable device 100. In the latter embodiment, the controller of the DCAmay transmit the depth information to the controller 135 of the wearabledevice 100.

The sensor device 115 generates one or more measurements signals inresponse to motion of the wearable device 100. The sensor device 115 maybe located on a portion of the frame 105 of the wearable device 100. Thesensor device 115 may include a position sensor, an inertial measurementunit (IMU), or both. Some embodiments of the wearable device 100 may ormay not include the sensor device 115 or may include more than onesensor device 115. In embodiments in which the sensor device 115includes an IMU, the IMU generates IMU data based on measurement signalsfrom the sensor device 115. Examples of sensor devices 115 include: oneor more accelerometers, one or more gyroscopes, one or moremagnetometers, another suitable type of sensor that detects motion, atype of sensor used for error correction of the IMU, or some combinationthereof. The sensor device 115 may be located external to the IMU,internal to the IMU, or some combination thereof.

Based on the one or more measurement signals, the sensor device 115estimates a current position of the wearable device 100 relative to aninitial position of the wearable device 100. The estimated position mayinclude a location of the wearable device 100 and/or an orientation ofthe wearable device 100 or the user's head wearing the wearable device100, or some combination thereof. The orientation may correspond to aposition of each ear relative to the reference point. In someembodiments, the sensor device 115 uses the depth information and/or theabsolute positional information from a DCA to estimate the currentposition of the wearable device 100. The sensor device 115 may includemultiple accelerometers to measure translational motion (forward/back,up/down, left/right) and multiple gyroscopes to measure rotationalmotion (e.g., pitch, yaw, roll). In some embodiments, an IMU rapidlysamples the measurement signals and calculates the estimated position ofthe wearable device 100 from the sampled data. For example, the IMUintegrates the measurement signals received from the accelerometers overtime to estimate a velocity vector and integrates the velocity vectorover time to determine an estimated position of a reference point on thewearable device 100. The reference point is a point that may be used todescribe the position of the wearable device 100. While the referencepoint may generally be defined as a point in space, however, in practicethe reference point is defined as a point within the wearable device100.

The audio system tracks motion of sound sources and dynamically updatesacoustic transfer functions. The audio system comprises a microphonearray, a controller, and a speaker array. However, in other embodiments,the audio system may include different and/or additional components.Similarly, in some cases, functionality described with reference to thecomponents of the audio system can be distributed among the componentsin a different manner than is described here. For example, some or allof the functions of the controller may be performed by a remote server.

The microphone arrays record sounds within a local area of the wearabledevice 100. A local area is an environment surrounding the wearabledevice 100. For example, the local area may be a room that a userwearing the wearable device 100 is inside, or the user wearing thewearable device 100 may be outside and the local area is an outside areain which the microphone array is able to detect sounds. The microphonearray comprises a plurality of acoustic detection locations that arepositioned on the wearable device 100. An acoustic detection locationincludes either an acoustic sensor or a port. A port is an aperture inthe frame 105 of the wearable device 100. In the case of an acousticdetection location, the port provides a coupling point for sound from alocal area to an acoustic waveguide that guides the sounds to anacoustic sensor. An acoustic sensor captures sounds emitted from one ormore sound sources in the local area (e.g., a room). Each acousticsensor is configured to detect sound and convert the detected sound intoan electronic format (analog or digital). The acoustic sensors may beacoustic wave sensors, microphones, sound transducers, or similarsensors that are suitable for detecting sounds.

In the illustrated configuration, the microphone array comprises aplurality of acoustic detection locations on the wearable device 100,for example acoustic detection locations 120 a, 120 b, 120 c, 120 d, 120e, and 120 f. The acoustic detection locations may be placed on anexterior surface of the wearable device 100, placed on an interiorsurface of the wearable device 100, separate from the wearable device100 (e.g., part of some other device), or some combination thereof. Insome embodiments, one or more of the acoustic detection locations 120a-f may also be placed in an ear canal of each ear. The configuration ofthe acoustic detection locations of the microphone array may vary fromthe configuration described with reference to FIG. 1. The number and/orlocations of acoustic detection locations may be different from what isshown in FIG. 1. For example, the number of acoustic detection locationsmay be increased to increase the amount of audio information collectedand the sensitivity and/or accuracy of the information. The acousticdetection locations may be oriented such that the microphone array isable to detect sounds in a wide range of directions surrounding the userwearing the wearable device 100. Each detected sound may be associatedwith a frequency, an amplitude, a phase, a time, a duration, or somecombination thereof.

The speaker array presents audio content based on the ATFs. The speakerarray comprises a plurality of acoustic emission locations on thewearable device 100. An acoustic emission location is a location of aspeaker or a port in the frame 105 of the wearable device 100. In thecase of an acoustic emission location, the port provides an outcouplingpoint of sound from an acoustic waveguide that separates a speaker ofthe speaker array from the port. Sound emitted from the speaker travelsthrough the acoustic waveguide and is then emitted by the port into thelocal area.

In the illustrated embodiment, the speaker array includes acousticemission locations 125 a, 125 b, 125 c, 125 d, 125 e, and 125 f. Inother embodiments, the speaker array may include a different number ofacoustic emission locations (more or less) and they may be placed atdifferent locations on the frame 105. For example, the speaker array mayinclude speakers that cover the ears of the user (e.g., headphones orearbuds). In the illustrated embodiment, the acoustic emission locations125 a-125 f are placed on an exterior surface (i.e., a surface that doesnot face the user) of the frame 105. In alternate embodiments some orall of the acoustic emission locations may be placed on an interiorsurface (a surface that faces the user) of the frame 105. Increasing thenumber of acoustic emission locations may improve an accuracy (e.g.,where a sound source is located) and/or resolution (e.g., a minimumdistance between discrete sound sources) of a sound scene analysisassociated with the audio content.

In some embodiments, each acoustic detection location is substantiallycollocated with a corresponding acoustic emission location.Substantially collocated refers to each acoustic detection locationbeing less than a quarter wavelength away from the correspondingacoustic emission location. The number and/or locations of acousticdetection locations and corresponding acoustic emission locations may bedifferent from what is shown in FIG. 1. For example, the number ofacoustic detection locations and corresponding acoustic emissionlocations may be increased to increase accuracy of a sound sceneanalysis.

The controller 135 processes information from the microphone array thatdescribes sounds detected by the microphone array. For each detectedsound, the controller 135 performs a DoA estimation. The DoA estimate isan estimated direction from which the detected sound arrived at anacoustic sensor of the microphone array. If a sound is detected by atleast two acoustic sensors of the microphone array, the controller 135can use the known positional relationship of the acoustic sensors andthe DoA estimate from each acoustic sensor to estimate a source locationof the detected sound, for example, via triangulation. The controller135 may use acoustic transfer functions to perform the DoA estimation.The accuracy of the source location estimation may increase as thenumber of acoustic sensors that detected the sound increases and/or asthe distance between the acoustic sensors that detected the soundincreases.

In some embodiments, the controller 135 may receive position informationof the wearable device 100 from a system external to the wearable device100. The position information may include a location of the wearabledevice 100, an orientation of the wearable device 100 or the user's headwearing the wearable device 100, or some combination thereof. Theposition information may be defined relative to a reference point. Theorientation may correspond to a position of each ear relative to thereference point. Examples of systems include an imaging assembly, aconsole (e.g., as described in FIG. 5), a simultaneous localization andmapping (SLAM) system, a depth camera assembly, a structured lightsystem, or other suitable systems. In some embodiments, the wearabledevice 100 may include sensors that may be used for SLAM calculations,which may be carried out in whole or in part by the controller 135. Thecontroller 135 may receive position information from the systemcontinuously or at random or specified intervals.

Based on parameters of the detected sounds, the controller 135 generatesone or more acoustic transfer functions associated with the audiosystem. The acoustic transfer functions may be array transfer functions(ATFs), head-related transfer functions (HRTFs), other types of acoustictransfer functions, or some combination thereof. An ATF characterizeshow the microphone array receives a sound from a point in space.Specifically, the ATF defines the relationship between parameters of asound at its source location and the parameters at which the microphonearray detected the sound. Parameters associated with the sound mayinclude frequency, amplitude, duration, a DoA estimate, etc. In someembodiments, at least some of the acoustic sensors of the microphonearray are coupled to an NED that is worn by a user. The ATF for aparticular source location relative to the microphone array may differfrom user to user due to a person's anatomy (e.g., ear shape, shoulders,etc.) that affects the sound as it travels to the person's ears.Accordingly, the ATFs of the microphone array are personalized for eachuser wearing the NED.

The HRTF characterizes how an ear receives a sound from a point inspace. The HRTF for a particular source location relative to a person isunique to each ear of the person (and is unique to the person) due tothe person's anatomy (e.g., ear shape, shoulders, etc.) that affects thesound as it travels to the person's ears. For example, in FIG. 1, thecontroller 135 may generate two HRTFs for the user, one for each ear. AnHRTF or a pair of HRTFs can be used to create audio content thatincludes sounds that seem to come from a specific point in space.Several HRTFs may be used to create surround sound audio content (e.g.,for home entertainment systems, theater speaker systems, an immersiveenvironment, etc.), where each HRTF or each pair of HRTFs corresponds toa different point in space such that audio content seems to come fromseveral different points in space. In some embodiments, the controller135 may update a pre-existing acoustic transfer function based on theDoA estimate of each detected sound. As the position of the wearabledevice 100 changes within the local area, the controller 135 maygenerate a new acoustic transfer function or update a pre-existingacoustic transfer function accordingly.

In some embodiments, the controller may perform DoA estimations, trackmovement of the sound sources, isolate the signals from different soundsources, and classify the sound sources. Operations of the controllerare described in detail below regarding FIGS. 3 and 4.

In the illustrated configuration the audio system is embedded into a NEDworn by a user. In alternate embodiments, the audio system may beembedded into a head-mounted display (HMD) worn by a user. Although thedescription above discusses the audio assemblies as embedded intoheadsets worn by a user, it would be obvious to a person skilled in theart that the audio assemblies could be embedded into different wearabledevices which could be worn by users elsewhere or operated by userswithout being worn.

FIG. 2A illustrates a wearable device 200 analyzing a sound scene 235within a local area 205, in accordance with one or more embodiments. Thewearable device 200 is worn by a user 210 and includes an audio system(e.g., as described in FIGS. 1 and 3-5). The local area 205 includes aplurality of sound sources, specifically, a person 215, a person 220, aperson 225, and a fan 230. The wearable device 200 performs a soundscene analysis. A sound scene describes, e.g., acoustic transferfunctions associated with sound sources, a number of sound sources,locations of the sound sources, movement of the sound sources,classifications of the sound sources, or some combination thereof.

The wearable device 200 estimates a DoA for each sound source. Dependingon the resolution of the wearable device 200 and the relative locationsof the sound sources, multiple sound sources may be grouped together asa single sound source for analysis by the wearable device 200. Forexample, the person 215 and the person 220 are located adjacent to eachother, and the wearable device 200 may, at least initially, identify theperson 215 and the person 220 as a single sound source.

Based on the DoA estimates, the wearable device 200 forms one or morebeams in the direction of each detected sound source, as furtherdescribed with respect to FIG. 3. To form a beam (also referred to asbeamforming) is a processing technique that the wearable device 200 usesto isolate and/or separate sounds produced by a sound source in thelocal area from other sound sources within the local area. For example,the wearable device 200 forms beam 241 around fan 230, beam 242 aroundperson 215 and person 220, and beam 243 around person 225. By forming abeam for each sound source, the wearable device may separately processthe data received by the microphone array for each sound source. Thewearable device 200 may increase the relative difference of audiosignals received from within a beam relative to other sounds in thelocal area 205. For example, the wearable device 200 may increase theamplitude of audio signals that are received from within a beam, maysuppress audio signals that are received from outside of the beam, orsome combination thereof.

The wearable device 200 is configured to classify each sound source. Forexample, based on the characteristics of the sound source, the wearabledevice 200 may classify a sound source as a human, an animal, anappliance, a vehicle, etc. The different classifications may affect howthe wearable device 200 processes the sounds received by the microphonearray and output by the speaker array. Based on the tracking, thebeamforming, the sound classification, or some combination thereof, theaudio system generates and/or updates sound filters, and provides thesound filters to the speaker array. The speaker array uses the soundfilters to present audio content. In some embodiments, to increase theability of the user to hear conversation, the wearable device 200 mayapply sound filters to increase the audio signal from beams with a soundsource classified as human, and the wearable device 200 may apply soundfilters to suppress the audio signal from beams with a sound sourceclassified as non-human.

FIG. 2B illustrates the wearable device 200 analyzing the sound scene235 after the person 225 has moved relative to the wearable device 200.The wearable device 200 is configured to monitor and analyze the soundscene 235 over time. As the person 225 moves, the wearable device 200may track the movement of the person 225. In some embodiments, thewearable device 200 may detect the movement based on a changing DoA ofthe sound source, visual information received by the wearable device200, or information received from an external data source. As relativepositioning between the wearable device 200 and one or more of thepersons 215, 220, 225 changes, the audio system dynamically adjusts thelocation of the beams to continue to include the persons 215, 220, 225.For example, as the person 225 walks towards the persons 215, 225, thewearable device 200 dynamically updates the sound scene analysis suchthat the beam 243 moves with the person 225. The wearable device 200 mayutilize the results of the tracking, beamforming, and classifying of thesound sources as feedback to evaluate the accuracy of the acoustictransfer functions generated by the wearable device 200. The wearabledevice 200 may update the acoustic transfer functions based on thefeedback. The updated acoustic transfer functions may be used to improvethe accuracy of the DoA estimation, tracking, beamforming, andclassifying. The updated acoustic transfer functions may be used toupdate the sound filters provided to the speaker array.

FIG. 3 is a block diagram of an audio system 300, in accordance with oneor more embodiments. The audio system in FIGS. 1, 2A, and 2B may beembodiments of the audio system 300. The audio system 300 detects soundto generate one or more acoustic transfer functions for a user. Theaudio system 300 may then use the one or more acoustic transferfunctions to generate audio content for the user. In the embodiment ofFIG. 3, the audio system 300 includes a microphone array 310, a speakerarray 320, and a controller 330. Some embodiments of the audio system300 have different components than those described here. Similarly, insome cases, functions can be distributed among the components in adifferent manner than is described here.

The microphone array 310 detects sounds within a local area surroundingthe microphone array 310. The microphone array 310 may include aplurality of acoustic sensors that each detect air pressure variationsof a sound wave and convert the detected sounds into an electronicformat (analog or digital). The plurality of acoustic sensors may bepositioned on an eyewear device (e.g., wearable device 100), on a user(e.g., in an ear canal of the user), on a neckband, or some combinationthereof. Each acoustic sensor of the microphone array 310 may be active(powered on) or inactive (powered off). The acoustic sensors areactivated or deactivated in accordance with instructions from thecontroller 330. In some embodiments, all of the acoustic sensors in themicrophone array 310 may be active to detect sounds, or a subset of theplurality of acoustic sensors may be active. An active subset includesat least two acoustic sensors of the plurality of acoustic sensors. Anactive subset may include, e.g., every other acoustic sensor, apre-programmed initial subset, a random subset, or some combinationthereof.

The speaker array 320 is configured to transmit sound to or from a user.The speaker array 320 may operate according to commands from thecontroller 330 and/or based on an audio characterization configurationfrom the controller 330. Based on the audio characterizationconfiguration, the speaker array 320 may produce binaural sounds thatseem to come from a particular point in space. The speaker array 320 mayprovide a sequence of sounds and/or surround sound to the user. In someembodiments, the speaker array 320 and the microphone array 310 may beused together to provide sounds to the user. In some embodiments, thespeaker array 320 may project sounds to specific locations in a soundscene, or the speaker array 320 may prevent sounds from being projectedto specific locations in a sound scene. The speaker array 320 maypresent sounds according to sound filters utilized by the controller330.

The speaker array 320 may be coupled to a wearable device to which themicrophone array 310 is coupled. In alternate embodiments, the speakerarray 320 may be a plurality of speakers surrounding a user wearing themicrophone array 310. In one embodiment, the speaker array 320 transmitstest sounds during a calibration process of the microphone array 310.The controller 330 may instruct the speaker array 320 to produce testsounds and then may analyze the test sounds received by the microphonearray 310 to generate acoustic transfer functions for the wearabledevice. Multiple test sounds with varying frequencies, amplitudes,durations, or sequences can be produced by the speaker array 320.

The controller 330 processes information from the microphone array 310.In addition, the controller 330 controls other modules and devices ofthe audio system 300. In the embodiment of FIG. 3, the controller 330includes the DoA estimation module 340, the transfer function module350, the tracking module 360, the beamforming module 370, theclassifying module 380, the sound filter module 385, and the personalassistant module 390.

The DoA estimation module 340 is configured to perform a DoA estimationfor detected sounds. If a sound is detected by at least two acousticsensors of the microphone array, the controller 330 can use thepositional relationship of the acoustic sensors and the DoA estimatefrom each acoustic sensor to estimate a source location of the detectedsound, for example, via triangulation. The estimated source location maybe a relative position of the source location in the local area relativeto a position of the microphone array 310. The position of themicrophone array 310 may be determined by one or more sensors on awearable device having the microphone array 310. In some embodiments,the controller 330 may determine an absolute position of the sourcelocation if an absolute position of the microphone array 310 is known inthe local area. The position of the microphone array 310 may be receivedfrom an external system (e.g., an imaging assembly, an AR or VR console,a SLAM system, a depth camera assembly, a structured light system etc.).The external system may create a virtual model of the local area, inwhich the local area and the position of the microphone array 310 aremapped. The received position information may include a location and/oran orientation of the microphone array in the mapped local area. Thecontroller 330 may update the mapping of the local area with determinedsource locations of detected sounds. The controller 330 may receiveposition information from the external system continuously or at randomor specified intervals.

The DoA estimation module 340 selects the detected sounds for which itperforms a DoA estimation. The DoA estimation module 340 populates anaudio data set with information. The information may include a detectedsound and parameters associated with each detected sound. Exampleparameters may include a frequency, an amplitude, a duration, a DoAestimate, a source location, a time of the measurement, or somecombination thereof. Each audio data set may correspond to a differentsource location relative to the microphone array 310 and include one ormore sounds having that source location. The DoA estimation module 340may populate the audio data set as sounds are detected by the microphonearray 310. The DoA estimation module 340 may evaluate the storedparameters associated with each detected sound and determine if one ormore stored parameters meet a corresponding parameter condition. Forexample, a parameter condition may be met if a parameter is above orbelow a threshold value or falls within a target range. If a parametercondition is met, the DoA estimation module 340 performs a DoAestimation for the detected sound. For example, the DoA estimationmodule 340 may perform a DoA estimation for detected sounds that have afrequency within a frequency range, an amplitude above a thresholdamplitude, a duration below a threshold duration range, other similarvariations or some combination thereof. Parameter conditions may be setby a user of the audio system 300, based on historical data, based on ananalysis of the information in the audio data set (e.g., evaluating thecollected information for a parameter and setting an average), or somecombination thereof. The DoA estimation module 340 may further populateor update the audio data set as it performs DoA estimations for detectedsounds. The DoA estimation module 340 may calculate a confidence levelfor each DoA estimate. The confidence level may be measured based on thesharpness of a peak in an underlying spatial spectrum. In someembodiments where a time difference of arrival-based algorithm isemployed, the confidence level may be measured based on a sharpness of across-correlation function. The confidence level for a DoA estimate mayrepresent a likelihood that the sound source is located in the locationestimated by the DoA estimation module 340. For example, the confidencelevel may range from 1-100, where a theoretical confidence level of 100represents that there is zero uncertainty in the DoA estimate, and aconfidence level of 1 represents a high level of uncertainty in the DoAestimate.

The transfer function module 350 is configured to generate one or moreacoustic transfer functions associated with the source locations ofsounds detected by the microphone array 310. Generally, a transferfunction is a mathematical function giving a corresponding output valuefor each possible input value. Each acoustic transfer function may beassociated with a position (i.e., location and/or orientation) of themicrophone array or person and may be unique to that position. Forexample, as the location of a sound source and/or a location ororientation of the microphone array or head of the person changes,sounds may be detected differently in terms of frequency, amplitude,etc. In the embodiment of FIG. 3, the transfer function module 350 usesthe information in the audio data set to generate the one or moreacoustic transfer functions. The information may include a detectedsound and parameters associated with each detected sound. The DoAestimates from the DoA estimation module 340 and their respectiveconfidence levels may be used as inputs to the transfer function module350 to improve the accuracy of the acoustic transfer functions.Additionally, the transfer function module 350 may receive feedback fromthe tracking module 360, the beamforming module 370, and the classifyingmodule 380 to update the acoustic transfer functions.

In some embodiments, the DoA estimation module 340 may preselect onlythe direct sound and remove the reflected sound. The direct sound can beused to extract the acoustic transfer function. For more informationregarding extracting acoustic transfer functions, see U.S. applicationSer. No. 16/015,879, entitled “AUDIO SYSTEM FOR DYNAMIC DETERMINATION OFPERSONALIZED ACOUSTIC TRANSFER FUNCTIONS” and filed on Jun. 22, 2018,the contents of which are incorporated by reference herein in theirentirety. The feedback can be used to control the adaptation process.

The feedback from the DoA estimation module 340, the tracking module360, the beamforming module 370, and the classifying module 380 may beused to update the acoustic transfer functions. Each module may beweighted differently. In some embodiments, the weight may be based onthe order in the processing chain. For example, the feedback from theDoA estimation module 340 may receive a weight of 0.4, the feedback fromthe tracking module 360 may receive a weight of 0.3, the feedback fromthe beamforming module 370 may receive a weight of 0.2, and the feedbackfrom the classifying module 380 may receive a weight of 0.1. However,this is just one example, and those skilled in the art will recognizethat many different weighting schemes may be used, and in someembodiments, the weights may be inferred by trial and error or byperforming a statistical analysis using experimental data.

The acoustic transfer functions may be used for various purposesdiscussed in greater detail below. In some embodiments, the transferfunction module 350 may update one or more pre-existing acoustictransfer functions based on the DoA estimates of the detected sounds. Asthe position (i.e., location and/or orientation) of the sound sources ormicrophone array 310 changes within the local area, the controller 330may generate a new acoustic transfer function or update a pre-existingacoustic transfer function accordingly associated with each position.

In some embodiments, the transfer function module 350 generates an arraytransfer function (ATF). The ATF characterizes how the microphone array310 receives a sound from a point in space. Specifically, the ATFdefines the relationship between parameters of a sound at its sourcelocation and the parameters at which the microphone array 310 detectedthe sound. The transfer function module 350 may generate one or moreATFs for a particular source location of a detected sound, a position ofthe microphone array 310 in the local area, or some combination thereof.Factors that may affect how the sound is received by the microphonearray 310 may include the arrangement and/or orientation of the acousticsensors in the microphone array 310, any objects in between the soundsource and the microphone array 310, an anatomy of a user wearing thewearable device with the microphone array 310, or other objects in thelocal area. For example, if a user is wearing a wearable device thatincludes the microphone array 310, the anatomy of the person (e.g., earshape, shoulders, etc.) may affect the sound waves as they travel to themicrophone array 310. In another example, if the user is wearing awearable device that includes the microphone array 310 and the localarea surrounding the microphone array 310 is an outside environmentincluding buildings, trees, bushes, a body of water, etc., those objectsmay dampen or amplify the amplitude of sounds in the local area.Generating and/or updating an ATF improves the accuracy of the audioinformation captured by the microphone array 310.

In one embodiment, the transfer function module 350 generates one ormore HRTFs. An HRTF characterizes how an ear of a person receives asound from a point in space. The HRTF for a particular source locationrelative to a person is unique to each ear of the person (and is uniqueto the person) due to the person's anatomy (e.g., ear shape, shoulders,etc.) that affects the sound as it travels to the person's ears. Thetransfer function module 350 may generate a plurality of HRTFs for asingle person, where each HRTF may be associated with a different sourcelocation, a different position of the person wearing the microphonearray 310, or some combination thereof. In addition, for each sourcelocation and/or position of the person, the transfer function module 350may generate two HRTFs, one for each ear of the person. As an example,the transfer function module 350 may generate two HRTFs for a user at aparticular location and orientation of the user's head in the local arearelative to a single source location. If the user turns his or her headin a different direction, the transfer function module 350 may generatetwo new HRTFs for the user at the particular location and the neworientation, or the transfer function module 350 may update the twopre-existing HRTFs. Accordingly, the transfer function module 350generates several HRTFs for different source locations, differentpositions of the microphone array 310 in a local area, or somecombination thereof.

In some embodiments, the transfer function module 350 may use theplurality of HRTFs and/or ATFs for a user to provide audio content forthe user. The transfer function module 350 may generate an audiocharacterization configuration that can be used by the speaker array 320for generating sounds (e.g., stereo sounds or surround sounds). Theaudio characterization configuration is a function, which the audiosystem 300 may use to synthesize a binaural sound that seems to comefrom a particular point in space. Accordingly, an audio characterizationconfiguration specific to the user allows the audio system 300 toprovide sounds and/or surround sound to the user, or to project soundsto different locations in the sound scene. The audio system 300 may usethe speaker array 320 to provide the sounds. In some embodiments, theaudio system 300 may use the microphone array 310 in conjunction with orinstead of the speaker array 320. In one embodiment, the plurality ofATFs, plurality of HRTFs, and/or the audio characterizationconfiguration are stored on the controller 330. The tracking module 360is configured to track locations of one or more sound sources. Thetracking module 360 may compare current DoA estimates or soundparameters and compare them with a stored history of previous DoAestimates or sound parameters. In some embodiments, the audio system 300may recalculate DoA estimates on a periodic schedule, such as once persecond, or once per millisecond. The tracking module may compare thecurrent DoA estimates with previous DoA estimates, and in response to achange in a DoA estimate for a sound source, the tracking module 360 maydetermine that the sound source moved. In some embodiments, the trackingmodule 360 may detect a change in location based on visual informationreceived by the wearable device or information received from an externaldata source. The tracking module 360 may track the movement of one ormore sound sources over time. The tracking module 360 may store valuesfor the number of sound sources and the location of each sound source ateach point in time. In response to a change in a value of the number orlocations of the sound sources, the tracking module 360 may determinethat a sound source moved. The tracking module 360 may calculate anestimate of the localization variance. The localization variance may beused as a confidence level for each determination of a change inmovement.

The beamforming module 370 is configured to form beams in the directionof sounds received at the microphone array 310 from discrete soundsources. The beamforming module 370 may isolate the audio signalreceived from within a beam from other sound sources in the local areabased on different DoA estimates from the DoA estimation module 340 andthe tracking module 360. Beamforming, also referred to as spatialfiltering, is a signal processing technique used in sensor arrays fordirectional reception. The beamforming module 370 may combine elementsin the microphone array 310 or the speaker array 320 in such a way thatsignals received from particular angles experience constructiveinterference while others experience destructive interference. To changethe directionality of the array, the beamforming module may control thephase and relative amplitude of the signal at each microphone orspeaker, in order to create a pattern of constructive and destructiveinterference in the wavefront. When analyzing sounds detected by themicrophone array 310, the beamforming module 370 may combine informationfrom different microphones in a way where the expected pattern ofradiation is preferentially observed. The beamforming module 370 maythus selectively analyze discrete sound sources in the local area. Insome embodiments, the beamforming module 370 may enhance the signal froma sound source. For example, the beamforming module 370 may apply soundfilters which eliminate signals above, below, or between certainfrequencies. Signal enhancement acts to enhance sounds associated with agiven identified sound source relative to other sounds detected by themicrophone array 310.

The beamforming module 370 may calculate a confidence level for theaccuracy of the location or other aspects of the beam. In someembodiments, the beamforming module 370 may use an array gaincalculation as the confidence level. The array gain is a ratio betweenan output signal to noise ratio (SNR) to an input SNR. A relativelyhigher array gain represents a higher confidence level. The beamformingmodule 370 may provide the isolated signals from the sound sources andtheir respective confidence levels to the transfer function module 350to be used as inputs to improve the accuracy of the acoustic transferfunctions.

The classifying module 380 is configured to classify the detected soundsources. In some embodiments, the classifying module 380 classifies theidentified sound sources as being either a human type or a non-humantype. A human type sound source is a person and/or device controlled bya person (e.g., a phone, a conferencing device, a telecommuting robot).A non-human type sound source is any sound source that is not classifiedas a human type sound source. A non-human type sound source may include,e.g., a television, a radio, an air conditioning unit, a fan, any soundsource that is not classified as a human type sound source, or somecombination thereof. In some embodiments, the classifying module 380classifies the sound source into narrower categories, such as male,female, dog, television, vehicle, etc. The classifying module 380 maystore a classification library. The classification library may store alist of sound source classifications, as well as parameters whichindicate that a sound source meets a particular classification. Forexample, sound source classifications may include: human, animal,mechanical, digital, instrument, vehicle, etc. In some embodiment, thesound source classifications may include sub-classifications. Forexample, the human classification may include the sub-classifications ofmale, female, adult, child, speaking, laughing, yelling, etc. Theparameters may include categories such as frequency, amplitude,duration, etc. Each classification or sub-classification is associatedwith parameters representing the classification. The classifying module380 may compare the parameters of a sound source with those in theclassification library to classify the sound source.

Additionally, in some embodiments, the user may manually classifyobjects and/or people in the local area. For example, the user mayidentify a person as a human using an interface on the wearable device.Once a sound source is classified, the classifying module 380 associatesthe acoustic transfer functions associated with the sound source asbeing of the same type.

The classifying module 380 determines a type of the sound source byanalyzing the acoustic transfer functions associated with the identifiedsound source and/or sounds detected by the microphone array 310. In someembodiments, the classifying module 380 may analyze the isolated signalsas provided by the beamforming module 370 to classify the sound sources.

The classifying module 380 may calculate a confidence level for theclassification of the sound sources. The classification module mayoutput a number that represents a probability that the input audiosample belongs to a given class. The probability number may be used asthe confidence level. The classifying module 380 may provide theclassification of the sound sources and their respective confidencelevels to the transfer function module 350 to be used as inputs toimprove the accuracy of the acoustic transfer functions.

The audio system 300 is continually receiving sounds from the microphonearray 310. Accordingly, the controller 330 can dynamically update (e.g.,via the modules within the controller 330) the acoustic transferfunctions and sound scene analysis as relative locations change betweenthe wearable device and any sound sources within the local area. Theupdated acoustic transfer functions may be used by the DoA estimationmodule 340, the tracking module 360, the beamforming module 370, and theclassifying module 380 to increase the accuracy of the respectivecalculations of each module.

The sound filter module 385 determines sound filters for the speakerarray 320. In some embodiments, the sound filter module 385 and thebeamforming module 370 may utilize binaural beamforming, which combinesbeamforming and playback into a single step using the acoustic transferfunctions. In such cases, the sound filter module 385 and thebeamforming module 370 determine the sound filters by applying anoptimization algorithm to the acoustic transfer functions. However, insome embodiments, the beamforming module 370 applies the optimizationalgorithm to the acoustic transfer functions prior to the sound filtermodule 385 determining the sound filters. The optimization algorithm issubject to one or more constraints. A constraint is a requirement thatcan affect the results of the optimization algorithm. For example, aconstraint may be, e.g., a classification of a sound source, that audiocontent output by the speaker array 320 is provided to ears of the user,energy and/or power of a sum of the acoustic transfer functionsclassified as human type is minimized or maximized, that audio contentoutput by the speaker array 320 has distortion less than a thresholdamount at the ears of the user, some other requirement that can affectthe results of the optimization algorithm, or some combination thereof.The optimization algorithm may be, e.g., a linearly constrained minimumvariance (LCMV) algorithm, a minimum variance distortionless response(MVDR), or some other adaptive beamforming algorithm that determinessound filters. In some embodiments, the optimization algorithm may alsoutilize a direction of arrival of sound from the identified soundsources and/or relative locations of the one or more sound sources tothe headset to determine the sound filters. The optimization algorithmmay output sound filters. The sound filter module 385 provides the soundfilters to the speaker array 320. The sound filters, when applied to anaudio signal, cause the speaker array 320 to present audio content thatamplifies or dampens sound sources. In some embodiments, the soundfilters may cause the speaker array 320 to amplify human sound sources,and to dampen non-human sound sources. In some embodiments, the soundfilters may cause the speaker array 320 to generate a sound field withreduced amplitudes in one or more damped regions that are occupied bysound sources.

As noted above, the optimization algorithm can be constrained by aclassification type of a sound source. For example, the sound filtermodule 385 and/or the beamforming module 370 may apply the optimizationalgorithm to the acoustic transfer functions in a manner such that anenergy of a sum of energies of the acoustic transfer functionsclassified as human type is minimized. An optimization algorithmconstrained in this manner may generate sound filters such that dampedareas would be located where sound sources classified as human type arepresent, but would not be located where sound sources classified asnon-human type are present. One advantage of classification is that itcan potentially reduce a number of damped regions within the soundfield, thereby reducing complexity of the sound field and hardwarespecifications for the speaker array 320 (e.g., a number of acousticemission locations and acoustic detection locations). Reduction in thenumber of damped regions may also increase suppression of the dampedregions used.

In response to the transfer function module 350 updating the acoustictransfer functions, the sound filter module 385 may apply theoptimization algorithm to the updated acoustic transfer functions. Thesound filter module 385 may provide the updated sound filters to thespeaker array 320. Having classified some or all sound sources in thesound scene, the sound filters may be applied to emphasize or suppressselected sound sources. The selected sound sources may be decided basedon a given scenario, a user's input, or various algorithms employed bythe device as described herein.

The personal assistant module 390 is configured to provide usefulinformation about the sound scene analysis to the user. The personalassistant module 390 may provide the information to the user via thespeaker array 320 or a visual display on a wearable device. For example,the personal assistant module 390 may provide the number, location, andclassification of the various sound sources to the user. The personalassistant module 390 may transcribe speech from a human sound source.The personal assistant module 390 may provide descriptive informationabout a sound source, such as information about a specific person ifthat person is listed in the classification library, or a make and modelof a mechanical sound source.

Additionally, the personal assistant module 390 may provide a predictiveanalysis of the sound scene. For example, the personal assistant module390 may determine that, based on the spatial information provided by thetracking module 360, a sound source identified by the classifying module380 as a vehicle is rapidly moving in the direction of the user, and thepersonal assistant module 390 may generate a notification of themovement of the vehicle to warn the user via the speaker array 320 or avisual display that the user is in danger of being struck by thevehicle. In some embodiments, the personal assistant module 390 maypredict or request input from the user regarding which sound sourcesshould be amplified and which sound sources should be dampened. Forexample, the personal assistant module 390 may determine, based onpreviously stored interactions with the user or with other users, thatthe sound from the closest human sound source to the user should beamplified, and all other sound sources should be damped. This may assistthe user in holding a conversation in a loud environment. Those skilledin the art will recognize that the above specific examples represent asmall portion of the many available uses for the personal assistantmodule 390 and audio system 300.

FIG. 4 is a flowchart illustrating a process 400 of generating andupdating acoustic transfer functions for a wearable device (e.g.,wearable device 100) including an audio system (e.g., audio system 300),in accordance with one or more embodiments. In one embodiment, theprocess of FIG. 4 is performed by components of the audio system. Otherentities may perform some or all of the steps of the process in otherembodiments (e.g., a console or a remote server). Likewise, embodimentsmay include different and/or additional steps, or perform the steps indifferent orders.

The audio system detects 410 sounds from one or more sound sources in alocal area surrounding the wearable device. In some embodiments, theaudio system stores the information associated with each detected soundin an audio data set.

In some embodiments, the audio system estimates a position of thewearable device in the local area. The estimated position may include alocation of the wearable device and/or an orientation of the wearabledevice or a user's head wearing the wearable device, or some combinationthereof. In one embodiment, the wearable device may include one or moresensors that generate one or more measurement signals in response tomotion of the wearable device. The audio system may estimate a currentposition of the wearable device relative to an initial position of thewearable device. In another embodiment, the audio system may receiveposition information of the wearable device from an external system(e.g., an imaging assembly, an AR or VR console, a SLAM system, a depthcamera assembly, a structured light system, etc.).

The audio system estimates 420 one or more acoustic transfer functionsassociated with the detected sounds. The acoustic transfer function maybe an array transfer function (ATF) or a head-related transfer function(HRTF). Accordingly, each acoustic transfer function is associated witha different source location of a detected sound, a different position ofa microphone array, or some combination thereof. As a result, the audiosystem may estimate a plurality of acoustic transfer functions for aparticular source location and/or position of the microphone array inthe local area.

The audio system performs 430 a Direction of Arrival (DoA) estimationfor each detected sound relative to the position of the wearable device.The DoA estimate may be represented as a vector between an estimatedsource location of the detected sound and the position of the wearabledevice within the local area. In some embodiments, the audio system mayperform a DoA estimation for detected sounds associated with a parameterthat meets a parameter condition. For example, a parameter condition maybe met if a parameter is above or below a threshold value or fallswithin a target range. The wearable device may calculate a confidencelevel for each DoA estimate. For example, the confidence level may rangefrom 1-100, where a theoretical confidence level of 100 represents thatthere is zero uncertainty in the DoA estimate, and a confidence level of1 represents a high level of uncertainty in the DoA estimate. Based onthe DoA estimates and the confidence levels for the DoA estimates, theaudio system may update the acoustic transfer functions.

The audio system detects 440 a change in location of one or more soundsources. The audio system may store a history of previously estimatedDoAs. In some embodiments, the audio system may recalculate DoAs on aperiodic schedule, such as once per second, or once per millisecond. Theaudio system may compare the current DoAs with previous DoAs, and inresponse to a change in a DoA for a sound source, the audio system maydetermine that the sound source moved. In some embodiments, the wearabledevice may detect a change in location based on visual informationreceived by the wearable device or information received from an externaldata source. The audio system may track the movement of one or moresound sources over time. The wearable device may calculate a confidencelevel for each determination of a change in movement. Based on thetracking of the sound sources and the confidence levels for the changesin location, the audio system may update the acoustic transferfunctions.

If the position of the microphone array changes within the local area,the audio system may generate one or more new acoustic transferfunctions or update one or more pre-existing acoustic transfer functionsaccordingly.

The audio system forms beams 450 in the directions of different soundsources. For example, the audio system may utilize a beamforming processto separate the signals from different sound sources for furtheranalysis. The audio system may analyze and process the sound receivedfrom each beam independently. The audio system may enhance the signalreceived from each beam. The audio system may calculate confidencelevels for the beamforming process and use the isolated signals from thesound sources and their respective confidence levels to update theacoustic transfer functions.

The audio system may classify 460 the sound sources. The audio systemmay compare the signals received from the sound sources with signalsassociated with known classifications. For example, the audio system mayclassify a sound source as a human based on a similarity tocharacteristics of a human classification in a classification library.The audio system may calculate confidence levels for the classificationand use the classifications of the sound sources and their respectiveconfidence levels to update the acoustic transfer functions.

The audio system may present 470 sound content using the speaker array.Based on the tracking, the beamforming, and the sound classification,the audio system generates and/or updates sound filters, and providesthe sound filters to the speaker array. The speaker array uses the soundfilters to present audio content. The sound filters may cause thespeaker array to amplify some sounds and suppress others. The specificuses for amplification and suppression may cover any desired purpose.For example, the sound filters may cause the speaker array to amplifythe sounds from a sound source that is identified as a human soundsource, while suppressing sounds from a sound source identified as anuisance sound source, such as a fan; the sound filters may suppressspeech and amplify white noise to reduce distraction while at work, thesound filters may amplify sound of an approaching vehicle to warn theuser; the sound filters may amplify the sound of a crying baby to drawattention; etc.

The audio system may adjust 480 the acoustic transfer functions. Theaudio system may adjust the acoustic transfer functions based on atleast one of the DoA estimates, the changes in location of soundsources, the isolation of the sound sources, or the classification ofthe sound sources. Additionally, the audio system may use the confidencelevels for the respective inputs to adjust the acoustic transferfunctions. The transfer function module adjusts the acoustic transferfunctions by combining the current/known acoustic transfer functionswith new/fresh acoustic transfer functions extracted from the mostrecent piece of audio signal. The acoustic transfer functions arecombined with certain weights, which may be selected based on theconfidence levels from the various modules. The weights may be directlyproportional to the overall confidence received from all other modules.For example, high confidence implies that the current acoustic transferfunctions are accurate, hence adaptation can be slow or stopped, whichmeans a high weight (e.g. greater than 0.5) may be assigned to the knownacoustic transfer function component, and low weight (e.g. less than0.5) may be assigned to the new data. In contrast, if the combinedconfidence is low for the current acoustic transfer functions, rapidadaptation may be required, in which case a high weight may be assignedto the acoustic transfer functions extracted from the recent audio data.

In some embodiments, the audio system may update the acoustic transferfunctions at any point throughout the process 400. The updated acousticfunctions may be used to perform DoA estimations, track the soundsources, form beams for the sound sources, identify the sound sources,provide sound filters to the speaker array, and present audio content.

The process 400 may be continuously repeated as a user wearing themicrophone array (e.g., coupled to an NED) moves through the local area,or the process 400 may be initiated upon detecting sounds via themicrophone array. By using the results of the steps of process 400 asfeedback which may be inputs for the estimation of the acoustic transferfunctions, the acoustic transfer functions and the overall performanceof the audio system and wearable device may be continuously improved.

Example of an Artificial Reality System

FIG. 5 is a system environment of a wearable device 505 including anaudio system 510, in accordance with one or more embodiments. The system500 may operate in an artificial reality environment. The system 500shown by FIG. 5 comprises a wearable device 505 and an input/output(I/O) interface 515 that is coupled to a console 501. The wearabledevice 505 may be an embodiment of the wearable device 100. While FIG. 5shows an example system 500 including one wearable device 505 and oneI/O interface 515, in other embodiments, any number of these componentsmay be included in the system 500. For example, there may be multiplewearable devices 505 each having an associated I/O interface 515 witheach wearable device 505 and I/O interface 515 communicating with theconsole 501. In alternative configurations, different and/or additionalcomponents may be included in the system 500. Additionally,functionality described in conjunction with one or more of thecomponents shown in FIG. 5 may be distributed among the components in adifferent manner than described in conjunction with FIG. 5 in someembodiments. For example, some or all of the functionality of theconsole 501 is provided by the wearable device 505.

The wearable device 505 presents content to a user comprising augmentedviews of a physical, real-world environment with computer-generatedelements (e.g., two dimensional (2D) or three dimensional (3D) images,2D or 3D video, sound, etc.). The wearable device 505 may be an eyeweardevice or a head-mounted display. In some embodiments, the presentedcontent includes audio content that is presented via the audio system300 that receives audio information (e.g., an audio signal) from thewearable device 505, the console 501, or both, and presents audiocontent based on the audio information.

The wearable device 505 includes the audio system 510, a depth cameraassembly (DCA) 520, an electronic display 525, an optics block 530, oneor more position sensors 535, and an inertial measurement unit (IMU)540. The electronic display 525 and the optics block 530 is oneembodiment of the lens 110 of FIG. 1. The position sensors 535 and theIMU 540 is one embodiment of sensor device 115 of FIG. 1. Someembodiments of the wearable device 505 have different components thanthose described in conjunction with FIG. 5. Additionally, thefunctionality provided by various components described in conjunctionwith FIG. 5 may be differently distributed among the components of thewearable device 505 in other embodiments, or be captured in separateassemblies remote from the wearable device 505.

The audio system 510 detects sound to generate one or more acoustictransfer functions for a user. The audio system 510 may then use the oneor more acoustic transfer functions to generate audio content for theuser. The audio system 510 may be an embodiment of the audio system 300.As described with regards to FIG. 3, the audio system 510 may include amicrophone array, a controller, and a speaker array, among othercomponents. The microphone array detects sounds within a local areasurrounding the microphone array. The microphone array may include aplurality of acoustic sensors that each detect air pressure variationsof a sound wave and convert the detected sounds into an electronicformat (analog or digital). The controller performs a DoA estimation forthe sounds detected by the microphone array. Based in part on the DoAestimates of the detected sounds and parameters associated with thedetected sounds, the controller generates one or more acoustic transferfunctions associated with the source locations of the detected sounds.The acoustic transfer functions may be ATFs, HRTFs, other types ofacoustic transfer functions, or some combination thereof. The controllermay generate instructions for the speaker array to emit audio contentthat seems to come from several different points in space. The audiosystem 510 may track the locations of the sounds, form beams around thelocations of the sounds, and classify the sounds. The results of thetracking, beamforming, and classifying, as well as any associatedconfidence levels, may be input to the controller to update the acoustictransfer functions.

The DCA 520 captures data describing depth information of a localenvironment surrounding some or all of the wearable devices 505. The DCA520 may include a light generator (e.g., structured light and/or a flashfor time-of-flight), an imaging device, and a DCA controller that may becoupled to both the light generator and the imaging device. The lightgenerator illuminates a local area with illumination light, e.g., inaccordance with emission instructions generated by the DCA controller.The DCA controller is configured to control, based on the emissioninstructions, operation of certain components of the light generator,e.g., to adjust an intensity and a pattern of the illumination lightilluminating the local area. In some embodiments, the illumination lightmay include a structured light pattern, e.g., dot pattern, line pattern,etc. The imaging device captures one or more images of one or moreobjects in the local area illuminated with the illumination light. TheDCA 520 can compute the depth information using the data captured by theimaging device or the DCA 520 can send this information to anotherdevice such as the console 501 that can determine the depth informationusing the data from the DCA 520.

In some embodiments, the audio system 510 may utilize the depthinformation which may aid in identifying directions of one or morepotential sound sources, depth of one or more sound sources, movement ofone or more sound sources, sound activity around one or more soundsources, or any combination thereof.

The electronic display 525 displays 2D or 3D images to the user inaccordance with data received from the console 501. In variousembodiments, the electronic display 525 comprises a single electronicdisplay or multiple electronic displays (e.g., a display for each eye ofa user). Examples of the electronic display 525 include: a liquidcrystal display (LCD), an organic light emitting diode (OLED) display,an active-matrix organic light-emitting diode display (AMOLED),waveguide display, some other display, or some combination thereof.

In some embodiments, the optics block 530 magnifies image light receivedfrom the electronic display 525, corrects optical errors associated withthe image light, and presents the corrected image light to a user of thewearable device 505. In various embodiments, the optics block 530includes one or more optical elements. Example optical elements includedin the optics block 530 include: a waveguide, an aperture, a Fresnellens, a convex lens, a concave lens, a filter, a reflecting surface, orany other suitable optical element that affects image light. Moreover,the optics block 530 may include combinations of different opticalelements. In some embodiments, one or more of the optical elements inthe optics block 530 may have one or more coatings, such as partiallyreflective or anti-reflective coatings.

Magnification and focusing of the image light by the optics block 530allows the electronic display 525 to be physically smaller, weigh less,and consume less power than larger displays. Additionally, magnificationmay increase the field of view of the content presented by theelectronic display 525. For example, the field of view of the displayedcontent is such that the displayed content is presented using almost all(e.g., approximately 110 degrees diagonal), and in some cases, all ofthe user's field of view. Additionally, in some embodiments, the amountof magnification may be adjusted by adding or removing optical elements.

In some embodiments, the optics block 530 may be designed to correct oneor more types of optical error. Examples of optical error include barrelor pincushion distortion, longitudinal chromatic aberrations, ortransverse chromatic aberrations. Other types of optical errors mayfurther include spherical aberrations, chromatic aberrations, or errorsdue to the lens field curvature, astigmatisms, or any other type ofoptical error. In some embodiments, content provided to the electronicdisplay 525 for display is pre-distorted, and the optics block 530corrects the distortion when it receives image light from the electronicdisplay 525 generated based on the content.

The IMU 540 is an electronic device that generates data indicating aposition of the wearable device 505 based on measurement signalsreceived from one or more of the position sensors 535. A position sensor535 generates one or more measurement signals in response to motion ofthe wearable device 505. Examples of position sensors 535 include: oneor more accelerometers, one or more gyroscopes, one or moremagnetometers, another suitable type of sensor that detects motion, atype of sensor used for error correction of the IMU 540, or somecombination thereof. The position sensors 535 may be located external tothe IMU 540, internal to the IMU 540, or some combination thereof. Inone or more embodiments, the IMU 540 and/or the position sensor 535 maybe monitoring devices capable of monitoring responses of the user toaudio content provided by the audio system 300.

Based on the one or more measurement signals from one or more positionsensors 535, the IMU 540 generates data indicating an estimated currentposition of the wearable device 505 relative to an initial position ofthe wearable device 505. For example, the position sensors 535 includemultiple accelerometers to measure translational motion (forward/back,up/down, left/right) and multiple gyroscopes to measure rotationalmotion (e.g., pitch, yaw, and roll). In some embodiments, the IMU 540rapidly samples the measurement signals and calculates the estimatedcurrent position of the wearable device 505 from the sampled data. Forexample, the IMU 540 integrates the measurement signals received fromthe accelerometers over time to estimate a velocity vector andintegrates the velocity vector over time to determine an estimatedcurrent position of a reference point on the wearable device 505.Alternatively, the IMU 540 provides the sampled measurement signals tothe console 501, which interprets the data to reduce error. Thereference point is a point that may be used to describe the position ofthe wearable device 505. The reference point may generally be defined asa point in space or a position related to the eyewear device's 505orientation and position.

The I/O interface 515 is a device that allows a user to send actionrequests and receive responses from the console 501. An action requestis a request to perform a particular action. For example, an actionrequest may be an instruction to start or end capture of image or videodata, or an instruction to perform a particular action within anapplication. The I/O interface 515 may include one or more inputdevices. Example input devices include: a keyboard, a mouse, a handcontroller, or any other suitable device for receiving action requestsand communicating the action requests to the console 501. An actionrequest received by the I/O interface 515 is communicated to the console501, which performs an action corresponding to the action request. Insome embodiments, the I/O interface 515 includes an IMU 540, as furtherdescribed above, that captures calibration data indicating an estimatedposition of the I/O interface 515 relative to an initial position of theI/O interface 515. In some embodiments, the I/O interface 515 mayprovide haptic feedback to the user in accordance with instructionsreceived from the console 501. For example, haptic feedback is providedwhen an action request is received, or the console 501 communicatesinstructions to the I/O interface 515 causing the I/O interface 515 togenerate haptic feedback when the console 501 performs an action. TheI/O interface 515 may monitor one or more input responses from the userfor use in determining a perceived origin direction and/or perceivedorigin location of audio content.

The console 501 provides content to the wearable device 505 forprocessing in accordance with information received from one or more of:the wearable device 505 and the I/O interface 515. In the example shownin FIG. 5, the console 501 includes an application store 550, a trackingmodule 555 and an engine 545. Some embodiments of the console 501 havedifferent modules or components than those described in conjunction withFIG. 5. Similarly, the functions further described below may bedistributed among components of the console 501 in a different mannerthan described in conjunction with FIG. 5.

The application store 550 stores one or more applications for executionby the console 501. An application is a group of instructions, that whenexecuted by a processor, generates content for presentation to the user.Content generated by an application may be in response to inputsreceived from the user via movement of the wearable device 505 or theI/O interface 515. Examples of applications include: gamingapplications, conferencing applications, video playback applications, orother suitable applications.

The tracking module 555 calibrates the system environment 500 using oneor more calibration parameters and may adjust one or more calibrationparameters to reduce error in determination of the position of thewearable device 505 or of the I/O interface 515. Calibration performedby the tracking module 555 also accounts for information received fromthe IMU 540 in the wearable device 505 and/or an IMU 540 included in theI/O interface 515. Additionally, if tracking of the wearable device 505is lost, the tracking module 555 may re-calibrate some or all of thesystem environment 500.

The tracking module 555 tracks movements of the wearable device 505 orof the I/O interface 515 using information from the one or more positionsensors 535, the IMU 540, the DCA 520, or some combination thereof. Forexample, the tracking module 555 determines a position of a referencepoint of the wearable device 505 in a mapping of a local area based oninformation from the wearable device 505. The tracking module 555 mayalso determine positions of the reference point of the wearable device505 or a reference point of the I/O interface 515 using data indicatinga position of the wearable device 505 from the IMU 540 or using dataindicating a position of the I/O interface 515 from an IMU 540 includedin the I/O interface 515, respectively. Additionally, in someembodiments, the tracking module 555 may use portions of data indicatinga position or the wearable device 505 from the IMU 540 to predict afuture position of the wearable device 505. The tracking module 555provides the estimated or predicted future position of the wearabledevice 505 or the I/O interface 515 to the engine 545. In someembodiments, the tracking module 555 may provide tracking information tothe audio system 300 for use in generating the sound filters.

The engine 545 also executes applications within the system environment500 and receives position information, acceleration information,velocity information, predicted future positions, or some combinationthereof, of the wearable device 505 from the tracking module 555. Basedon the received information, the engine 545 determines content toprovide to the wearable device 505 for presentation to the user. Forexample, if the received information indicates that the user has lookedto the left, the engine 545 generates content for the wearable device505 that mirrors the user's movement in a virtual environment or in anenvironment augmenting the local area with additional content.Additionally, the engine 545 performs an action within an applicationexecuting on the console 501 in response to an action request receivedfrom the I/O interface 515 and provides feedback to the user that theaction was performed. The provided feedback may be visual or audiblefeedback via the wearable device 505 or haptic feedback via the I/Ointerface 515.

Additional Configuration Information

The foregoing description of the embodiments of the disclosure has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the disclosure be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

What is claimed is:
 1. A method comprising: estimating acoustic transferfunctions associated with sounds from one or more sound sources in alocal area; classifying, based on a classification library, a soundsource in the one or more sound sources; calculating a first confidencelevel for the classifying; calculating a second confidence level for abeamforming process for the sound source; and updating the acoustictransfer functions based in part on the first confidence level and thesecond confidence level.
 2. The method of claim 1, further comprisingisolating a signal from the sound source from other sound sources in thelocal area of the wearable device.
 3. The method of claim 1, furthercomprising estimating a direction of arrival (DoA) of the sound source.4. The method of claim 3, further comprising calculating a thirdconfidence level for the DoA estimate, and a fourth confidence level fora tracking estimate.
 5. The method of claim 4, further comprisingupdating the acoustic transfer functions based on at least one of thethird confidence level or the fourth confidence level.
 6. The method ofclaim 1, further comprising: storing values for the number and locationsof the one or more sound sources over time; and detecting a change in atleast one of the number or the locations.
 7. The method of claim 1,further comprising: updating sound filters based in part on the updatedacoustic transfer functions; and presenting audio content based on theupdated sound filters.
 8. An audio system comprising: a microphone arrayconfigured to detect sounds from one or more sound sources in a localarea of the audio system; and a controller configured to: estimateacoustic transfer functions associated with the sounds; classify, basedon a classification library, a sound source in the one or more soundsources; calculate a first confidence level for the classifying;calculate a second confidence level for a beamforming process for thesound source; and update the acoustic transfer functions based in parton the first confidence level and the second confidence level.
 9. Theaudio system of claim 8, wherein the controller is further configured toisolate a signal from the sound source from other sound sources in thelocal area of the wearable device.
 10. The audio system of claim 8,wherein the controller is further configured to estimate a direction ofarrival (DoA) of the sound source.
 11. The audio system of claim 10,wherein the controller is further configured to calculate a thirdconfidence level for the DoA estimate, and a fourth confidence level fora tracking estimate.
 12. The audio system of claim 11, wherein thecontroller is further configured to update the acoustic transferfunctions based on at least one of the third confidence level or thefourth confidence level.
 13. The audio system of claim 8, wherein thecontroller is further configured to: store values for the number andlocations of the one or more sound sources over time; and detect achange in at least one of the number or the locations.
 14. The audiosystem of claim 8, wherein the controller is further configured to:update sound filters based in part on the updated acoustic transferfunctions; and present audio content based on the updated sound filters.15. The audio system of claim 8, wherein the controller is furtherconfigured to generate a notification of a movement of the sound source.16. A non-transitory computer-readable storage medium comprisinginstructions executable by a processor, the instructions when executedcausing the processor to perform actions comprising: estimating acoustictransfer functions associated with sounds from one or more sound sourcesin a local area; classifying, based on a classification library, a soundsource in the one or more sound sources; calculating a first confidencelevel for the classifying; calculating a second confidence level for abeamforming process for the sound source; and updating the acoustictransfer functions based in part on the first confidence level and thesecond confidence level.
 17. The non-transitory computer-readablestorage medium of claim 16, the actions further comprising isolating asignal from the sound source from other sound sources in the local areaof the wearable device.
 18. The non-transitory computer-readable storagemedium of claim 16, the actions further comprising estimating adirection of arrival (DoA) of the sound source.
 19. The non-transitorycomputer-readable storage medium of claim 18, the actions furthercomprising calculating a third confidence level for the DoA estimate,and a fourth confidence level for a tracking estimate.
 20. Thenon-transitory computer-readable storage medium of claim 16, the actionsfurther comprising: updating sound filters based in part on the updatedacoustic transfer functions; and presenting audio content based on theupdated sound filters.